Currently this is a draft: there are some clear open questions to answer before it is ready to merge

• Do we want to cover chars 0-32 as filecontents does?
• Do we want a more complex approach to a preceding optional argument. For example, fancyvrb sets the catcode of ^^M prior to the \@ifnextchar to ensure it is tokenizes in a recognisable way
• Do we want to auto-apply ! to any preceding optional arg
• Is the letter f the best choice? I can see an argument for c ('collected'), x (charcode of b + v), w (v-squaredv), ...

I'd maybe drop the warning on +f and make f just behave correctly. There is no good reason not to, imho, it's also not +b for multiline bodies of environments.

Is it possible in the architecture of ltcmd to change the behaviour of an argument grabber based on the next argument type? If so the optional argument should be searched with the full category code regime of f in place, and if it's found the regime should be reverted, the optional argument collected, the verbatim catcode regime reactivated and the body collected. And an optional argument should have otherwise the same observable rules as it would have with a b body (so should be found even with a new line between \begin{foo} and the optional argument).

Cf. the handling of the optional argument in enverb (which as the only strange point also finds the opening bracket of the optional argument after multiple blank lines -- maybe I should revise this :)).

As for the argument name, what about an upper case V?

I'd maybe drop the warning on +f and make f just behave correctly. There is no good reason not to, imho, it's also not +b for multiline bodies of environments.

Thinking here is that reading the arg. spec. should be clear: collecting an environment will allow newlines, so should be +f not just f. That's why I've currently made it a warning - at a technical level, it's not an issue.

As for the argument name, what about an upper case V?

I thought about that - but the pattern we have at the moment is all uppercase specs relate to the matching lowercase in that they take an extra {<default>} argument. That's not the situation here, which is why at the moment I went with something else.

Is it possible in the architecture of ltcmd to change the behaviour of an argument grabber based on the next argument type? If so the optional argument should be searched with the full category code regime of f in place, and if it's found the regime should be reverted, the optional argument collected, the verbatim catcode regime reactivated and the body collected. And an optional argument should have otherwise the same observable rules as it would have with a b body (so should be found even with a new line between \begin{foo} and the optional argument).

Whilst we don't currently do this, it would be possible to do a 'pre-scan' to establish if special settings are needed. The issue I think is that unlike say fancyvrb or enverb, we don't know the nature of the optional argument(s) - they don't have to be delimited by [..]. So for example if someone has +d^$ +f, we need to be able to cope with that in a (somewhat) predictable way. We also have the case of multiple optional arguments, so it's all a bit more 'interesting' - in the end this could all be done, it's a question of what we want to provide and how well that fits more widely into the LaTeX landscape.

As for the argument name, what about an upper case V?

I thought about that - but the pattern we have at the moment is all uppercase specs relate to the matching lowercase in that they take an extra {<default>} argument. That's not the situation here, which is why at the moment I went with something else.

that pattern is strong and easy to understand so I don't think we should break it, thus that would rule V out in my opinion

Maybe a dumb Saturday evening idea, but what about using a special sort of processor? Right now >{...}is a processor that is applied after grabbing, but we could have <{...} for before, or less general c{named-catcode-regiment}. The you would have c{verbatim}b+ for proces the body with "verbatim" catcodes. And fancyvrb could define its own named set. The whole thing would then work also with other arg specifiers. Handwaving ...

Whilst we don't currently do this, it would be possible to do a 'pre-scan' to establish if special settings are needed. The issue I think is that unlike say fancyvrb or enverb, we don't know the nature of the optional argument(s) - they don't have to be delimited by [..]. So for example if someone has +d^$ +f, we need to be able to cope with that in a (somewhat) predictable way. We also have the case of multiple optional arguments, so it's all a bit more 'interesting' - in the end this could all be done, it's a question of what we want to provide and how well that fits more widely into the LaTeX landscape.

I would think this is bound to produce problems forever. Besides, what you do with respect to chars being %? Could be done, as you say but would be slow and probably riddled with errors in weird cases that we hadn't thought about.

Maybe a dumb Saturday evening idea, but what about using a special sort of processor? Right now >{...}is a processor that is applied after grabbing, but we could have <{...} for before, or less general c{named-catcode-regiment}. The you would have c{verbatim}b+ for proces the body with "verbatim" catcodes. And fancyvrb could define its own named set. The whole thing would then work also with other arg specifiers. Handwaving ...

I see the idea, but we have a few things that mean I suspect it's not quite right

• We need to fiddle with \endlinechar
• We may want to 'set and reset' the catcodes (see other replies)
• It leaves finding the end point a bit awkward
• There's the entire question of nesting

However, it does suggest something we could do - see parallel reply.

Whilst we don't currently do this, it would be possible to do a 'pre-scan' to establish if special settings are needed. The issue I think is that unlike say fancyvrb or enverb, we don't know the nature of the optional argument(s) - they don't have to be delimited by [..]. So for example if someone has +d^$ +f, we need to be able to cope with that in a (somewhat) predictable way. We also have the case of multiple optional arguments, so it's all a bit more 'interesting' - in the end this could all be done, it's a question of what we want to provide and how well that fits more widely into the LaTeX landscape.

I would think this is bound to produce problems forever. Besides, what you do with respect to chars being %? Could be done, as you say but would be slow and probably riddled with errors in weird cases that we hadn't thought about.

I think @Skillmon's approach could work, with a logic for seeking an optional arg:

1. Set all specials and ^^M verbatim
2. Reset the catcode of the peek token to normal
3. Peek ahead - loop to remove spaces/newlines if needed
4. Reset catcodes and grab the optional arg if found

This could be done by adding an additional specifier (lets say *) which means 'use a verbatim-safe approach to grabbing an optional arg'. That would allow us to avoid needed to unpick everything to decide if to use this slower approach.

Performance is not that bad, it's just \begingroup\expanded{\noexpand\dospecials\catcode\peekchar=\the\catcode\peekchar}\@ifnextchar\peekchar (sort of) and then, if the argument is found \endgroup and collect it, else grab the verbatim argument. And: To match current behaviour of ltcmd you should only strip the first newline (because two consecutives are normally a \par hence don't match the optional argument's start-token). And depending on the stripping of the first newline in the f-argument you need to conditionally add the ignored newline to the verbatim argument if no optional argument is found.

Performance is not that bad, it's just \begingroup\expanded{\noexpand\dospecials\catcode\peekchar=\the\catcode\peekchar}\@ifnextchar\peekchar (sort of) and then, if the argument is found \endgroup and collect it, else grab the verbatim argument. And: To match current behaviour of ltcmd you should only strip the first newline (because two consecutives are normally a \par hence don't match the optional argument's start-token). And depending on the stripping of the first newline in the f-argument you need to conditionally add the ignored newline to the verbatim argument if no optional argument is found.

I don't think we can use a group, and we have to deal with looping for spaces, but yes, I guess you are broadly right about performance.

And: To match current behaviour of ltcmd you should only strip the first newline (because two consecutives are normally a \par hence don't match the optional argument's start-token). And depending on the stripping of the first newline in the f-argument you need to conditionally add the ignored newline to the verbatim argument if no optional argument is found.

Good point about multiple ^^M. I suspect space skipping here is not desirable anyway. Perhaps then best to go with my * suggestion, which could also be applied to before v?

I'd maybe drop the warning on +f and make f just behave correctly. There is no good reason not to, imho, it's also not +b for multiline bodies of environments.

Thinking here is that reading the arg. spec. should be clear: collecting an environment will allow newlines, so should be +f not just f. That's why I've currently made it a warning - at a technical level, it's not an issue.

One might well argue that both b and f should act the same way, + or not - an easy adjustment if that's the overall agreement. If we do that, I'd normalise to one agreed form.

I don't think we can use a group, and we have to deal with looping for spaces, but yes, I guess you are broadly right about performance.

Of course you can use a group! You're doing right now for the f-argument as well. Again see enverb (use of \enverb@body@setup and \enverb@search@oarg). And yes, you need to loop, obviously, but it's a tight loop that only needs to check three tokens (active space, ^^M, and the optional argument's begin-token, and as I said it needs to keep track of the tokens it ignored and whether it already gobbled up a ^^M and re-add those if it doesn't find the optional argument).

Good point about multiple ^^M. I suspect space skipping here is not desirable anyway. Perhaps then best to go with my * suggestion, which could also be applied to before v?

Yes, applying * to v would then also be desirable. Though ltcmd could check for that itself I guess it's much less complicated to let the user decide and not let ltcmd peek ahead for the structure of (possibly all) following arguments.

One might well argue that both b and f should act the same way, + or not - an easy adjustment if that's the overall agreement. If we do that, I'd normalise to one agreed form.

My argumentation is that +v means multiple lines are allowed, that meaning doesn't make sense for f, as restricting an environment to a single line sounds rather odd. And +b means that b might take a \par token. That's not applicable to f as ^^M is set to other. So the only valid meaning of f vs. +f would be f not allowing empty/blank lines in the body and +f doing so. Still odd for verbatim material, but at least understandable. The current behaviour is just odd (if it still is the current behaviour, haven't taken another look at the changes).

Good point about multiple ^^M. I suspect space skipping here is not desirable anyway. Perhaps then best to go with my * suggestion, which could also be applied to before v?

Yes, applying * to v would then also be desirable. Though ltcmd could check for that itself I guess it's much less complicated to let the user decide and not let ltcmd peek ahead for the structure of (possibly all) following arguments.

I think I'll implement the suggestion, we can look at it then decide if we need automation - that would mainly sit in a different part of the code so would still require the underlying * code.

I've made two adjustments here:

• f-type are always 'long'
• Use @Skillmon's suggestion to grab optional arguments 'verb safe'

The second idea is partly-done. At present, you have to turn this on using a * specifier. If the mechanism looks OK, I can automate this - but probably only for an immediately-preceeding D-type argument. I'll do that, and document, once we have agreement this looks OK. See the test file for demos.

We could make life a little easier e.g. by saying that optional args before f-type have to use 'other' chars, or using a charcode peek test (more risky). But on balance, I'd rather go with an automated version of what I already have -

Other than the basic question 'is this the right letter', I think we are good-to-go here now.

I'll continue collecting typos here (because here I conveniently see what was added in this PR).

Neither @davidcarlisle nor @muzimuzhi seem to have reviewed this yet.
Not sure how important this is.

Neither @davidcarlisle nor @muzimuzhi seem to have reviewed this yet. Not sure how important this is.

I pinged a group of people to get some feedback - we'd taken a while and I felt most ideas were resolved, but needed to be sure about the implementation. So I was not expected all reviews to be 'in' before merged - we have time before a release for adjustment.